Using machine learning to quantify the risk of loneliness and social isolation on care home entry

Sam Rickman Jose-Luis Fernandez, Juliette Malley

October 2023

Overview

How can the analysis of unstructured, free text data from administrative care records inform long term care policy and practice?

  1. Paper 1: An algorithm to classify whether free text case records indicate loneliness or social isolation.
  2. Paper 2: Does the extracted information predict the risk of care home entry?

  3. Paper 3: Predicting risk of abuse from free text records.

Loneliness and care home entry

Why is this important?

Why is this important?

  1. Moving to a residential or nursing care home is associated with a loss of independence, dignity and privacy.1
  2. Loneliness is a modifiable risk factor.
  3. Services for loneliness have changed. Publicly-funded day centre attendance has declined every year since 2012: from 15% to 6% in 2019.

What do we know?

 

  1. Loneliness and social isolation affect 20-34% of older people (WHO, 2021). 2
  2. As significant a predictor of mortality as smoking, obesity or hypertension (Holt-Lunstead et al., 2015). 3
  3. Policy concern in UK, e.g. Government Loneliness Strategy (2018), 4 and internationally e.g. WHO report (2021) report on social isolation and loneliness in older people. 5

 

 

Based on definitions in de Jong-Gierveld (1987) 6 Coyle & Dugan (2012) 7

What about care home entry?

A systematic review of the impact of loneliness on healthcare utilization found two good-fair quality papers internationally (Smith & Victor, 2021)8:

  1. Hanratty et al. (2018)9 Loneliness as a risk factor for care home admission in the English Longitudinal Study of Ageing.
    1. Strengths: Longitudinal, controls for confounders: age, health, dementia.
    2. Limitations: Two year gaps between waves. Follow up.
    3. Method: Logistic regression.
  1. Russell et al. (1997)10 Loneliness and nursing home admission among rural older adults.
    1. Strengths: Longitudinal, controls for age, health, dementia.
    2. Method: Logistic regression.
    3. Limitations: Generalisability in time and space.

Conceptual framework: how to fill the research gap


Production of Welfare (POW) approach. Adapted from Knapp (1984). 11

This study: the data

  1. Data from London Borough of Islington.
  2. All adults who were:
    • Aged 65 years and over by the 31st August 2020
    • Receiving care services in the community for at least a year since the end of 2015.
  3. In total 3,046 individuals. Data is pseudonymised using software which removes: names, ethnicity, locations, postcodes, dates, times, currencies, telephone numbers, email addresses.

 

What does the data look like?

Why not just add a social isolation or loneliness field?

  1. In our data the mean unique N structured questions per person is 482 across our extract of 9 forms over 10 years (11 versions of each form).
    • Social care workers know that faults exist and do not entirely trust records they retrieve.12
  2. Analyses of preventable deaths find workers are not familiar with the contents of their own agency records.13
  3. Direction of travel is away from bureaucratic, structured forms and towards “a return to the narrative”.14
  4. We already have a lot of free text.

Why not just add a social isolation or loneliness field?

  1. In Islington the mean unique number of structured questions per person is 482 (s.d. 175).
    • A 2018 study found workers did not fully trust records they retrieved knowing that faults were entered.15
  2. Analyses of preventable deaths find workers are not familiar with the contents of their own agency records.16
  3. Direction of travel is away from bureaucratic, structured forms and towards “a return to the narrative”.17
  4. There is a huge amount of free text currently in the records.

 

Training a classification model

Train test split

  • Train set: 10k sentences.
  • Test set: 3.6k sentences.
  • Binary classification (0 or 1).
  • Imbalanced data: ~90% negative class.
  • High inter-rater reliability: Cohen’s \(\kappa = 0.89\).

 

Language representation models

Three approaches:

  1. Frequency-based word vectors (e.g. count of each word in sentence).
  2. Pre-trained word vectors (e.g. GloVe).
  3. Context-dependent word vectors (transformer-based approaches e.g. RoBERTa, MiniLM).

Distributional semantics

“You shall know a word by the company it keeps” (J. R. Firth 1957: 11).18

Model 2: Pre-trained word vectors: GloVe 19

  • High dimensional vectors.
  • Trained on huge volumes of text, e.g. all English books ever, Wikipedia, internet forums, newspapers.
  • Similar words closer together in vector space e.g. GloVe (Pennington et al. 2014)20 (2d mapping)

Model 3: RoBERTa (Liu, 2019) 21

  1. Words have different connotations depending on context:
    • social work: statutory services, care and support.
    • social media: internet, technology.
    • social life: friends, family, gatherings.

Results

Classification metrics

\[\begin{aligned} \textrm{precision (positive predictive value)} = \frac{TP}{TP + FP} \\ \\ \textrm{recall (true positive rate)} = \frac{TP}{TP + FN} \\ \\ \textrm{accuracy} = \frac{\textrm{N correct}}{\textrm{N classified}} \\ \\ F_1 = 2 \frac{\textrm{precision} \cdot \textrm{recall}}{\textrm{precision} + \textrm{recall}} \\ \\ \end{aligned}\]

\[ TP = \text{True Positive} \\ \\ TN = \text{True Negative} \\ \\ FP = \textrm{False Positive} \\ \\ FN = \text{False Negative} \\ \\ \text{recall} = \text{sensitivity} \]

Results

model type model classifier accuracy precision recall f1
Transformers RoBERTa Neural network 0.97 0.95 0.87 0.91
Transformers DistilRoBERTa Neural network 0.96 0.90 0.82 0.86
Transformers MiniLM Neural network 0.92 0.81 0.60 0.69
Pre-trained vectors GloVe Neural network 0.90 0.78 0.50 0.61
Word counts Bag of Words QDA 0.27 0.15 0.83 0.26

 

What about the rest of the data?

Classification of free text at time of first assessment

  • 10 year observation window contains first needs assessment for 1,331 individuals.
  • Initial case notes as within 90 days of initial assessment.
  • Classified just over 1m sentences:
    • Free text first assessment form (180k sentences)
    • Free text case notes (850k sentences)
  • Circumvents temporal question.

Results

Results: proportion lonely or isolated at time of first assessment

Construct validity regression

ELSA
Islington
CES-D UCLA Assessment Case notes Either Both
Age 1.01 (0.99-1.03) 1.00 (0.99-1.02) 1.01 (0.99-1.02) 1.01 (1.00-1.03) 1.01 (1.00-1.02) 1.01 (1.00-1.03)
Dressing 0.88 (0.62-1.23) 0.92 (0.66-1.28) 0.87 (0.78-0.97) ** 0.91 (0.82-1.01) . 0.88 (0.79-0.99) * 0.87 (0.78-0.97) *
Ethnicity (non-white) 1.60 (0.47-5.34) 1.33 (0.36-5.05) 0.96 (0.76-1.23) 0.98 (0.77-1.24) 0.99 (0.77-1.27) 0.95 (0.72-1.23)
Lives alone 6.12 (4.36-8.70) *** 3.57 (2.62-4.90) *** 1.37 (1.08-1.75) ** 1.38 (1.09-1.76) ** 1.52 (1.18-1.95) *** 1.39 (1.06-1.82) *
Meals 1.36 (0.89-2.07) 1.31 (0.86-1.99) 1.02 (0.90-1.16) 1.09 (0.96-1.23) 1.09 (0.95-1.24) 1.04 (0.91-1.19)
Memory 1.14 (0.47-2.71) 0.65 (0.26-1.57) 1.40 (1.25-1.57) *** 1.40 (1.25-1.57) *** 1.50 (1.33-1.70) *** 1.47 (1.30-1.67) ***
Mobility 0.86 (0.56-1.32) 1.19 (0.79-1.80) 0.89 (0.81-0.99) * 0.99 (0.90-1.10) 1.01 (0.90-1.13) 0.85 (0.76-0.96) **
Safety & risk 1.35 (0.55-3.26) 1.17 (0.48-2.85) 1.03 (0.92-1.15) 0.94 (0.84-1.05) 1.00 (0.89-1.12) 0.96 (0.85-1.09)
Sex (F) 0.92 (0.66-1.27) 0.99 (0.72-1.35) 1.15 (0.91-1.45) 0.89 (0.70-1.12) 0.97 (0.76-1.24) 1.06 (0.82-1.37)
Shopping 1.61 (1.13-2.29) ** 1.45 (1.03-2.05) * 1.09 (0.93-1.28) 1.02 (0.88-1.20) 1.06 (0.90-1.24) 1.07 (0.90-1.28)
Toileting 1.25 (0.79-1.97) 1.46 (0.94-2.26) . 1.04 (0.95-1.15) 0.88 (0.80-0.97) * 0.93 (0.84-1.03) 0.97 (0.87-1.08)
Unpaid care 1.84 (1.08-3.19) * 2.22 (1.27-3.98) ** 1.01 (0.78-1.30) 0.88 (0.68-1.14) 0.94 (0.71-1.23) 0.92 (0.70-1.23)
*** < 0.001; ** <0.01; * <0.05; . <0.1

Paper 2: Quantifying risk of care home entry

Overall approach

  1. Look at a person’s needs at the time of initial assessment with care services, where the individual is in the community \(N = 1331\).
  2. Model 1: Time-to-event model. Do loneliness or isolation affect how long is spent receiving care in the community?

\[h(t | \textbf{X}_i) = h_0(t) \cdot e^{\beta_i \textbf{X}_i}\]

  1. Model 2: Are people who are lonely or socially isolated receiving a different quantity of services? Generalised Estimating Equations (GEE) model.

\[ g\{(E(y_{it}) | \textbf{X})\} = \beta_{it} \textbf{X}_{it}, \: \: \: y \sim F \: \textrm{with parameters} \: \theta_{it} \]

Where \(y\) is service cost, \(\textbf{X}\) is a vector of covariates including loneliness and other needs.

Model inputs

Descriptive statistics

Category Value Overall (%) Care home (%) p
Lonely / Isolated Lonely/isolated 380 (29) 135 (38) <0.001
Not lonely/isolated 951 (71) 221 (62)
Sex Female 826 (62) 212 (60) 0.282
Male 505 (38) 144 (40)
Ethnicity Non-white 440 (33) 96 (27) 0.005
White 891 (67) 260 (73)
Age Group (60,75] 208 (16) 54 (15) <0.001
(75,80] 183 (14) 40 (11)
(80,85] 273 (21) 81 (23)
(85,90] 311 (23) 87 (24)
(90,95] 239 (18) 74 (21)
(95,110] 117 (9) 20 (6)
Personal Care Fully independent 161 (12) 45 (13) 0.012
Largely independent 173 (13) 86 (24)
Partial independence 304 (23) 69 (19)
Limited independence 288 (22) 78 (22)
High support needs 703 (53) 44 (12)
Very high support needs 87 (7) 34 (10)
Memory No issues 519 (39) 70 (20) <0.001
Mild 354 (27) 89 (25)
Some 269 (20) 101 (28)
Marked 157 (12) 76 (21)
Severe 32 (2) 20 (6)
Lives Alone FALSE 374 (28) 154 (43) 0.399
TRUE 957 (72) 202 (57)
Safety & Risk Less than daily 76 (6) 16 (4) <0.001
Telecare 260 (20) 55 (15)
Daily 561 (42) 121 (34)
More than daily 216 (16) 46 (13)
Someone nearby 181 (14) 99 (28)
1to1+ 37 (3) 19 (5)
Unpaid Care FALSE 374 (28) 121 (34) 0.005
TRUE 957 (72) 235 (66)
Day-to-Day Activities Largely independent 173 (13) 42 (12) 0.141
Some needs 455 (34) 110 (31)
High support needs 703 (53) 204 (57)

From survival to competing risks

Estimate \(h_k (t)\) for \(k \in \{1,2\}\), where \(1 = \text{care home}, 2 = \text{death}\).

  1. Survival hazard function:22 \[h (t) = \lim_{\Delta\to0} \frac{1}{\Delta(t)} P(t < T \leqslant t + \Delta(t) \: | \: T > t)\]

  2. Cause-specific hazard function:23 \[h^{\color{yellow}{cs}}_k (t) = \lim_{\Delta\to0} \frac{1}{\Delta(t)}P(t < T \leqslant t + \Delta(t), \: \mathbin{\color{yellow}{K = k}} \: | \: T > t)\]

  3. Subdistribution hazard function (Fine & Gray model):24 \[h^{\color{#33FFFF}{sd}}_k (t) = \lim_{\Delta\to0} \frac{1} {\Delta(t)}P(t < T \leqslant t + \Delta(t), \: \color{yellow}{K = k} \: | \\ \: T > t \: \mathbin{\color{#33FFFF}{ \cup \: (T \leqslant t \cap K \neq k) }}\color{white}{})\]

 

Cause-specific vs subdistribution hazard

Adapted from Mozmunder (2018) 25

Proportional hazards assumption

Estimate \(h_k^j (t)\) for \(k \in \{1,2\}\) and \(j \in {1,2}\), where \(k: {1 = \text{care home}, 2 = \text{death}}\) and \(j: {1 = \text{cause-specific}, 2 = \text{subdistribution}}.\) 26 27

\[ h_k^{j} (t | \textbf{X}) = h_{0,k}^{j} \cdot e^{\beta_{i,k}^{j} \textbf{X}_i} \]

  • This assumption is violated for memory, cognition, personal care, day-to-day activities and unpaid care. Solutions:
    1. Splines (Royston-Parmar model) with interaction between spline constant and time-dependent covariate, Stata stpm2cr.
    2. Stratify by time-dependent variable. R survival::coxph() (cause-specific hazard) and cmprsk::crr() (subdistribution hazard).

Preliminary results

Cumulative incidence by loneliness or isolation

Regression model including covariates

Hazard ratio: subdistribution hazard

Hazard ratio: cause-specific hazard

Discussion and policy implications

  1. Loneliness or isolation appear to increase risk of care home entry controlling for functional ability (physical and cognitive), unpaid care and living alone.
  2. Can commissioners account for this in preventative and targeting services?
  3. Loneliness is a modifiable risk factor. Effective interventions: befriending, social and lifestyle activities, psychosocial (individual and group-based), technological, occupational therapy.
  4. Are interventions cost-effective? Will risk of care home entry be affected if loneliness is modified?

Footnotes

  1. Bradshaw, S.A., Playford, E.D. and Riazi, A., 2012. Living well in care homes: a systematic review of qualitative studies. Age and ageing, 41(4), pp.429-440.

  2. World Health Organization, (2021). Social isolation and loneliness among older people: advocacy brief.

  3. Holt-Lunstad, J., Smith, T.B., Baker, M., Harris, T. and Stephenson, D., (2015). Loneliness and social isolation as risk factors for mortality: a meta-analytic review. Perspectives on psychological science, 10(2), pp.227-237.

  4. DCMS: Department for Culture, Media and Sport, (2018). A connected society: a strategy for tackling loneliness, TSO.

  5. World Health Organization, (2021). Social isolation and loneliness among older people: advocacy brief.

  6. Coyle, C.E. and Dugan, E., (2012). Social isolation, loneliness and health among older adults. Journal of aging and health, 24(8), pp.1346-1363.

  7. de Jong-Gierveld, J., (1987). Developing and testing a model of loneliness. Journal of personality and social psychology, 53(1), p.119.

  8. Smith, K.J. and Victor, C., (2021). The Association of Loneliness with Health and Social Care Utilization in Older Adults in the General Population: A Systematic Review. The Gerontologist.

  9. Hanratty, B., Stow, D., Collingridge Moore, D., Valtorta, N.K. and Matthews, F., (2018). Loneliness as a risk factor for care home admission in the English Longitudinal Study of Ageing. Age and ageing, 47(6), pp.896-900.

  10. Russell, D. W., Cutrona, C. E., de la Mora, A., & Wallace, R. B. (1997). Loneliness and nursing home admission among rural older adults. Psychology and aging, 12(4), 574. https://doi.org/10.1037/0882-7974.12.4.574

  11. Martin R J Knapp. The economics of social care. Macmillan International Higher Education, 1984. ISBN 1349177083.

  12. Lagsten, J. and Andersson, A., 2018. Use of information systems in social work–challenges and an agenda for future research. European Journal of Social Work, 21(6), pp.850-862.

  13. Preston-Shoot, Michael, 2019, Analysis of Safeguarding Adult Reviews April 2017 - March 2019, https://www.local.gov.uk/sites/default/files/documents/National%20SAR%20Analysis%20Final%20Report%20WEB.pdf

  14. Farelly, Nicola, 2023 Good practice in digitisation of social care records

  15. Lagsten, J. and Andersson, A., 2018. Use of information systems in social work–challenges and an agenda for future research. European Journal of Social Work, 21(6), pp.850-862.

  16. Preston-Shoot, Michael, 2019, Analysis of Safeguarding Adult Reviews April 2017 - March 2019, https://www.local.gov.uk/sites/default/files/documents/National%20SAR%20Analysis%20Final%20Report%20WEB.pdf

  17. Farelly, Nicola, 2023 Good practice in digitisation of social care records

  18. Firth, J.R., 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis.

  19. Pennington, J., Socher, R. and Manning, C.D., (2014), October. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).

  20. Pennington, J., Socher, R. and Manning, C.D., (2014), October. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).

  21. Pennington, J., Socher, R. and Manning, C.D., (2014), October. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).

  22. Austin, P.C., Lee, D.S. and Fine, J.P., 2016. Introduction to the analysis of survival data in the presence of competing risks. Circulation, 133(6), pp.601-609. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4741409/

  23. Austin, P.C., Lee, D.S. and Fine, J.P., 2016. Introduction to the analysis of survival data in the presence of competing risks. Circulation, 133(6), pp.601-609. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4741409/

  24. Mozumder, S.I., Rutherford, M.J. and Lambert, P.C., 2017. A flexible parametric competing-risks model using a direct likelihood approach for the cause-specific cumulative incidence function. The Stata Journal, 17(2), pp.462-489.

  25. Mozumder, S.I., Analysing competing risks data using flexible parametric survival models: what tools are available e in Stata, which ones to use and when?, 2018 London Stata Conference | 6 - 7 September 2018, https://www.stata.com/meeting/uk18/slides/uk18_Mozumder.pdf

  26. Mozumder, S.I., Rutherford, M.J. and Lambert, P.C., 2017. A flexible parametric competing-risks model using a direct likelihood approach for the cause-specific cumulative incidence function. The Stata Journal, 17(2), pp.462-489.

  27. Prentice, R.L., Kalbfleisch, J.D., Peterson Jr, A.V., Flournoy, N., Farewell, V.T. and Breslow, N.E., 1978. The analysis of failure times in the presence of competing risks. Biometrics, pp.541-554.